Advanced path prediction

ABSTRACT

The present disclosure provides systems and methods for mapping a determined path of travel. The path of travel may be mapped to a camera view of a camera affixed to a vehicle. In some embodiments, the path of travel may be mapped to another view that is based on a camera, such as a bird&#39;s eye view anchored to the camera&#39;s position at a given time. These systems and methods may determine the path of travel by incorporating data from later points in time.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.17/023,305, filed on the 16^(th) of Sep. 2020, which is a continuationof U.S. patent application Ser. No. 16/138,059 filed on the 21^(st) ofSep. 2018, and entitled “ADVANCED PATH PREDICITON,” which is acontinuation of and claims priority to PCT/US2017/23781, filed on the23^(rd) of Mar. 2017, and titled “ADVANCED PATH PREDICTION”, whichclaims the benefit of U.S. Provisional Patent Application No.62/312,445, filed on the 23^(rd) of Mar. 2016, and titled, “METHODS FORPREDICTING AND/OR TRAINING RESPONSES”, the disclosures of which areexpressly incorporated by reference in their entireties.

BACKGROUND Field

Certain aspects of the present disclosure generally relate tointelligent driving monitoring systems (IDMS), driver monitoringsystems, advanced driver assistance systems (ADAS), and autonomousdriving systems, and more particularly to systems and methods formapping a determined path of travel to a reference frame that is basedon a camera position.

Background

Driving a moveable object, such as a vehicle or a robot, depends onunderstanding the “rules of the road”, many of which are unwritten. Anoperator of a moving car, for example, may need to manage the proximityof other cars and objects, sometimes according to right-of-way rules,and sometimes according to generally accepted driving behaviors. Forexample, it may be desirable for a driver of a moving vehicle tomaintain a safe distance between his vehicle and other moving vehicle.Likewise, it may be desirable to avoid slow-moving or stationaryobjects.

For many driving situations, successful driving may depend on predictinga path of travel. For example, if a driver notices an object on a road,an appropriate response may be to change lanes. In another example, if adriver notices a slow-moving car, he may respond differently dependingon whether the slow-moving car is currently in the driver's lane. If so,he may need to slow down or modify his path of travel.

Reliable path prediction may benefit a number of driving relateddevices, including ADAS and IDMS devices. For example, a path predictionsystem may aid an IDMS in determining an extent to which a drivingbehavior is safe or unsafe. While much effort has been expended in pathprediction systems and methods, the complexity and variability ofreal-world driving situations continue to make path predictionchallenging.

Accordingly, aspects of the present disclosure are directed to improvedsystems and methods for path of travel prediction. Aspects of thepresent disclosure may be directed to improving systems and methods thatmay depend on path of travel prediction, such as IDMS, drivermonitoring, ADAS, and autonomous driving systems, among others.

SUMMARY OF THE INVENTION

The present disclosure provides systems and methods for mapping adetermined path of travel. In some embodiments, the path of travel maybe mapped to a camera view. In some embodiments, the path of travel maybe determined in response to other objects. Certain path predictionsystems and methods improve upon the prior art by incorporatinginformation from later points in time. In addition, systems and methodsfor detecting an object are provided that may process an image or videoframe faster and/or at higher resolution in comparison to currentmethods. In addition, methods are presented for detecting and/orrecognizing an object in a video or in a Lidar and/or radar data frame.In some embodiments, and object may be detected and/or recognized basedon the appearance of the object in a later data frame.

Certain aspects of the present disclosure provide a method of mapping adetermined path of travel. The method generally includes determining apath of travel of a vehicle from a first time to a second time, whereinthe second time is later than the first time; and mapping the determinedpath of travel to a first reference frame, wherein the first referenceframe is based on a position of a camera at the first time.

Certain aspects of the present disclosure provide an apparatusconfigured to map a determined path of travel. The apparatus generallyincludes a memory unit; and at least one processor coupled to the memoryunit, in which the at least one processor is configured to: determine apath of travel of a vehicle from a first time to a second time, whereinthe second time is later than the first time; and map the determinedpath of travel to a first reference frame, wherein the first referenceframe is based on a position of a camera at the first time, and whereinthe camera is attached to the vehicle.

Certain aspects of the present disclosure provide an apparatusconfigured to map a determined path of travel. The apparatus generallyincludes means for determining a path of travel of a vehicle from afirst time to a second time, wherein the second time is later than thefirst time; and means for mapping the determined path of travel to afirst reference frame, wherein the first reference frame is based on aposition of a camera at the first time, and wherein the camera isattached to the vehicle.

Certain aspects of the present disclosure provide a non-transitorycomputer-readable medium having program code recorded thereon formapping a determined path. The program code is executed by a processorand generally comprises program code to determine a path of travel of avehicle from a first time to a second time, wherein the second time islater than the first time; and map the determined path of travel to afirst reference frame, wherein the first reference frame is based on aposition of a camera at the first time, and wherein the camera isattached to the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a device configured to map a determinedpath of travel in accordance with certain aspects of the presentdisclosure.

FIG. 2 illustrates an example of a method for mapping a determined pathof travel in accordance with certain aspects of the present disclosure.

FIG. 3A illustrates an example of a method for mapping a determined pathof travel in accordance with certain aspects of the present disclosure.

FIG. 3B illustrates an example of a method for mapping a determined pathof travel in accordance with certain aspects of the present disclosure.

FIG. 3C illustrates an example of a method for mapping a determined pathof travel in accordance with certain aspects of the present disclosure.

FIG. 4 illustrates an example of a method for mapping a determined pathof travel in accordance with certain aspects of the present disclosure.

FIG. 5A illustrates an example of a method for mapping a determined pathof travel in accordance with certain aspects of the present disclosure.

FIG. 5B illustrates an example of a method for mapping a determined pathof travel in accordance with certain aspects of the present disclosure.

FIG. 5C illustrates an example of a method for mapping a determined pathof travel in accordance with certain aspects of the present disclosure.

DETAILED DESCRIPTION

The detailed description set forth below, in connection with theappended drawings, is intended as a description of variousconfigurations and is not intended to represent the only configurationsin which the concepts described herein may be practiced. The detaileddescription includes specific details for the purpose of providing athorough understanding of the various concepts. However, it will beapparent to those skilled in the art that these concepts may bepracticed without these specific details. In some instances, well-knownstructures and components are shown in block diagram form in order toavoid obscuring such concepts.

Based on the teachings, one skilled in the art should appreciate thatthe scope of the disclosure is intended to cover any aspect of thedisclosure, whether implemented independently of or combined with anyother aspect of the disclosure. For example, an apparatus may beimplemented or a method may be practiced using any number of the aspectsset forth. In addition, the scope of the disclosure is intended to coversuch an apparatus or method practiced using other structure,functionality, or structure and functionality in addition to or otherthan the various aspects of the disclosure set forth. It should beunderstood that any aspect of the disclosure disclosed may be embodiedby one or more elements of a claim.

The word “exemplary” is used herein to mean “serving as an example,instance, or illustration.” Any aspect described herein as “exemplary”is not necessarily to be construed as preferred or advantageous overother aspects.

Although particular aspects are described herein, many variations andpermutations of these aspects fall within the scope of the disclosure.Although some benefits and advantages of the preferred aspects arementioned, the scope of the disclosure is not intended to be limited toparticular benefits, uses or objectives. Rather, aspects of thedisclosure are intended to be broadly applicable to differenttechnologies, system configurations, networks and protocols, some ofwhich are illustrated by way of example in the figures and in thefollowing description of the preferred aspects. The detailed descriptionand drawings are merely illustrative of the disclosure rather thanlimiting, the scope of the disclosure being defined by the appendedclaims and equivalents thereof.

It must be noted that as used herein and in the appended claims, thesingular forms “a”, “an”, and “the” include plural referents unless thecontext clearly dictates otherwise. Thus, for example, reference to “anobject” includes a plurality of objects.

Path Prediction Challenges

Systems and methods for path prediction may rely on lane detection. Iflane boundaries are detected, a path of travel may be inferred based onthe detected lane boundaries. Such a system, for example, may predict apath of travel that maintains a currently observed distance to adetected lane boundary.

Current methods for determining a path of travel from a given point intime may include recording an image or video frame and using a computervision (CV) based lane detection. CV based lane detection may includeedge detection followed by curve fitting, such as with a Hough Transformor RANSAC. While this approach may provide acceptable path of travelpredictions in some situations, it may fail to provide acceptable pathof travel predictions in situations in which lane detection itself maybe challenging.

Challenging lane detection situations may include roads that are curved,roads that do not have clear lane markings, and dense traffic scenariosin which lane markings may be occluded by other cars. Examples of suchchallenging situations include side streets, parking lots, freshly pavedroads, as well as snow and rain covered roads. Additionally, even whenmarkings are present, the detection of markings may be difficult orconfusing, such as in intersections and turn bays, places where oldmarkings partially or fully remain, or in heavy traffic where other carsmay occlude the lane markings. Lastly, roads may be curvy, making itunclear if an object ahead of a car is in the same lane or in a nearbylane, and therefore whether a driver should adjust her path of travel.For current systems, therefore, path prediction remains a challengingproblem. While path prediction methods may operate satisfactorily insome situations, such as on un-crowded highways where lane markings areclear and present, they may fail to operate in other situations. Forexample, current path prediction systems may fail to operate when lanemarkings are unclear or not present.

Certain aspects of the present disclosure are directed to determiningthe path of travel in a way that may be robust to situations in whichlane markings are not clearly visible, are occluded, and the like, asdescribed below.

Some current methods of path prediction may rely on a machine learningmodel that has been trained to predict driving actions based on sensormeasurements. In some cases, a model may have access to up-to-the-momentdata, such as a recently captured video frame data, and may additionallyhave access to a history or sensor data. Based on sensor data, a machinelearning model may compute a prediction of a future path of travel.

As with methods based on lane detection, path prediction based onmachine learning may be confronted with many challenges that may hinderits utility. For example, machine learning methods may require largeamounts of labeled training data. For path prediction, obtaining labelsfor safe and/or appropriate paths of travel for a given situation may becumbersome or expensive. In addition, the dynamics and variability ofreal-world driving scenarios may limit the time horizon to which a pathmay be predicted within a desired accuracy.

Certain aspects of the present disclosure are directed to determiningthe path of travel in a way that may be used to facilitate thecollection of a desired amount of labeled training data, as describedbelow.

Desired Latency for Path of Travel Systems in IDMS

An IDMS may monitor aspects of the driver's driving behavior, which mayrefer to driving actions. An IDMS may, for example, make determinationsabout whether certain driving maneuvers are appropriate in a givensituation. An IDMS may also include a “driver” monitoring system thatmay monitor physiological signals (such as pupil diameter) of a driver.An IDMS may be applied to drivers of vehicles, such as cars, boats, andairplanes, as well as to factory robots, humanoid robots, and the like.Furthermore, an IDMS may be applied to moving objects being driven by ahuman driver or by an algorithmic controller. In some applications, anIDMS may determine a grade for a driver's performance or for theperformance of an algorithmic controller.

A time between an occurrence of an event and a time that a systemreports the occurrence of the event may be referred to as a latency. Alatency of less than one second may be referred to as a real-timelatency, or a system that is configured for real-time performance.Near-real time performance may correspond to a system that may makedeterminations at a rate that is slower than real-time, but that maystill affect the driver's driving. In some cases, near real-timeperformance may correspond to a latency between 1 second to 1 minute.

A desired latency for an autonomous driving system may be less than onesecond. Real-time performance for an autonomous driving system maycorrespond to a system that may make determinations about theenvironment within a time window that is short enough that the driver(or controller) may incorporate the determination in his (or its)response to the environment. Likewise, such ADAS or autonomous drivingsystems may be said to operate “in-the-loop.” “In-the-loop” may meanthat the ADAS operates within a typical sensory-to-motor response loopof a driver. For example, an ADAS may sense that another car is enteringthe lane of the driver and may alert the driver in less than one second,as described above. As another example, an ADAS may alert the driverthat a lane change would be unsafe at the current time.

Accordingly, an autonomous driving system may be expected tocontinuously or periodically respond to the surrounding environment. Assuch, methods have been developed in support of ADAS or autonomousdriving systems that may produce alerts or driving actions based oninformation that is available at the current time.

In some IDMS applications, a user's latency tolerance may be more thanone minute. Some IDMS, for example, may be configured so that they donot communicate the results of the monitoring in real-time or nearreal-time. In some embodiments, an IDMS may not communicate the resultsof the monitoring to the driver at all. Rather, the IDMS may communicatea driving behavior inference to a third-party, such a fleet safetymanager, who may have a latency tolerance of several minutes. In thesesituations, and others, an IDMS may not need to meet a real-time or nearreal-time reporting latency.

An IDMS may also operate in real-time or near-real-time. In one example,a near real-time system may make determinations about a driving statewhile the driver is still in the determined state. A combined ADAS andIDMS system running in near real-time, for example, may notify a driverthat his driving behavior is indicative of distracted driving. In somecases, near real-time performance may correspond to detecting abehavioral state of a driver within 5 seconds to 1 minute of its onset.

Certain aspects of the present disclosure may be applied to anIntelligent Driver Monitoring System (IDMS), combined ADAS and IDMSsystems, and the like. In addition, certain real-time systems may beenabled according to certain aspects of the present disclosure.

“Back-to-the-Future” Path of Travel Determination

Certain aspects of the present disclosure may exploit a reduced latencytolerance of an IDMS. Certain aspects may also be applied to low-latencysystems, such as ADAS and autonomous driving systems. Since an IDMS isnot necessarily “in-the-loop”, an analysis of a driving situation at agiven time need not be limited to information that is available at thattime. Rather, certain aspects of the present disclosure are directed tomethods that improve the analysis of a driving situation at a given timeby incorporating information from the “future”. Certain aspects may beapplied instead of, or in addition to, using CV approaches to determinea path of travel based on a “presently” received video frame. Accordingto certain aspects, “future” frames and/or sensor data may be used todetermine the “future” path of travel from the vantage point of the“presently” received video frame. Since a determination regarding adriving situation at a given time may be performed at a later time, the“future” information relative to that given time may have alreadyoccurred. This approach may be called a “Back-to-the-Future” approach.

FIG. 1 illustrates an embodiment of the aforementioned systems andmethods for mapping a determined path of travel. The device 100 mayinclude input sensors (which may include a forward facing camera 102, adriver facing camera 104, connections to other cameras that are notphysically mounted to the device, inertial sensors 106, car OBD-II portsensor and vehicle data (which may be obtained through a Bluetoothconnection 108), Lidar, radar, and the like) and compute capability 110.The compute capability may be a CPU or an integrated System-on-a-chip(SOC), which may include a CPU and other specialized compute cores, suchas a graphics processor (GPU), gesture recognition processor, and thelike. In some embodiments, a system or method for mapping a determinedpath of travel may include wireless communication to cloud services,such as with Long Term Evolution (LTE) 116 or Bluetooth 108communication to other devices nearby. In addition, LTE 116 or Bluetooth108 communication may be used to gather data from disaggregated sensors,which, in some embodiments may include Lidar, radar, and the like. Thedevice may also include a global positioning system (GPS) either as aseparate module 112, or integrated within a System-on-a-chip 110. Insome embodiments, a disaggregated sensor, such as Lidar, radar, or GPSmay communicate directly to the device 100, may communicate to thedevice via an intermediary processing unit, may communicate to thedevice via a cloud server, may communicate to a cloud server for furtherprocessing, possibly in combination with data communicated to the cloudfrom the device 100. For example, the cloud may provide real-timeanalytics assistance. In an embodiment involving cloud services, thecloud may facilitate aggregation and processing of data for offlineanalytics or for training of machine learning models. The device mayfurther include memory storage 114.

An exemplary embodiment may comprise a camera 102 mounted in a vehicle'swindshield and a means for determining the path of travel, which may bebased on camera 102 input. In some embodiments, the path of travel maybe determined based on visual odometry that is based on the camerainput, or based on visual inertial odometry (VIO) that is based at leastin part on camera and inertial sensor input. The path of travel may bedetermined by other means, such as via crowd-sourcing of likely pathsbased on previously recorded data, as described below, and/or usingnon-visual sensors such as GPS. The system may then map the determinedfuture path of travel to the perspective of the camera view at time to,or any other recorded camera view, or in some embodiments, a predictedcamera view at another time. For example, the determined path of travelmay be mapped to a camera view collected from another car, a camera viewof a stationary camera, or a camera view based on a 3D map of theenvironment.

Another exemplary embodiment may comprise a means for determining a pathor travel, which may include determining a path for travel from sensors,such as a camera 102 mounted in a car's windshield, a GPS receiver 112,inertial sensors 106, and other sensors. Alternatively, the means fordetermining a path of travel may include communicating data about thecurrent state of the car to a cloud server and receiving a typical pathof travel based on the communicated data. In an embodiment involving avehicle, the system may determine the following distance in seconds tothe closest car, if any, along the future path of travel based on thevehicle's velocity at a time to. In this example, the system may berunning in near-real time (e.g. at a 5 second latency), such that thetime t0 is 5 seconds in the past. The system may process the video frameat time t0. The processing may include detecting bounding boxes for carsin that video frame. The system may then determine the future path oftravel for the next 5 seconds after t0. That is, the system may thendetermine the future path of travel from the perspective of the frame att0 that includes the path of travel up to the second time. The secondtime may correspond to the time that the most recent sensor data wererecorded (such as the most recent GPS reading) in the time between t0and the end of the 5 second latency. From the perspective of t0, thepath of travel from t0 to the second time may be considered a “future”path of travel. Several means for determining the future path of travelare contemplated. The future path of travel may be based, at least inpart, on GPS data, future data frames, which may include video frames,GPS readings, Lidar and/or radar scans, (frames that were capturedbetween time t0 and the present time), and/or continuously orperiodically collected data from inertial sensors and the like.

A system enabled with certain aspects of the present disclosure may thenperform a mapping that may overlay the future path of travel on theimage that was collected at time t0. In some embodiments, the system maydetermine the closest car that impacts the determined path of travel,determine the distance to that car, and may convert that distance totime based on the ego-car's current velocity (the velocity of the car inwhich the system is mounted). In another exemplary embodiment, thedistance to another car may be determined based on Lidar and/or radarsensory systems. In one embodiment, the Lidar and/or radar may augmentcamera-based image processing. Alternatively, the Lidar and/or radar mayreplace camera-based image processing.

In addition to mapping the path of travel to a camera view, the path oftravel may also be mapped to a different reference frame that may beanchored to a camera position, such as an overhead view that is alignedwith a camera that is mounted in the moving vehicle.

FIG. 2 illustrates an example of mapping a path of travel in accordancewith certain aspects of the present disclosure. The vehicle for whichthe path is determined may be referred to as the ego-car. A satelliteimage 200 depicts a portion of a curved road. The pin marker 202indicates the location of an ego-car at a first time, time to. The pinmarker 204 indicates the location of the ego-car at time t1 (t0+1second). The pin marker 206 indicates the location of the ego-car attime t2 (t0+2 seconds). The pin marker 202 may be referred to as the“current” position in accordance with certain aspects of the presentdisclosure. The line 208 indicates the path of travel of the ego carfrom the first time, time t0, to a second time which may be severalseconds later.

In FIG. 3A, a mapping of the future path of travel 308 (corresponding tothe path of travel 208 illustrated in FIG. 2 ) is overlaid on the“current” frame 302. The “current” frame 302 corresponds to the position202 of the camera within the ego-car at time t0. The mapping comprises aprojection of the GPS path 304, illustrated in FIG. 3C, from ahorizontal plane 306 to an overhead (bird's-eye-view) reference frame310 illustrated in FIG. 3B, and then to the camera view 302 illustratedin FIG. 3A. The bird's eye view reference frame 310 is anchored to thecamera position at the bottom. The future path of travel 308,illustrated in FIG. 3B may be projected into the “current” frame 302with an inverse perspective mapping.

FIG. 3A further illustrates that a system may include timestamps 312 forthe current video frame 302 and timestamps for the current GPS reading314. The system may also include an internal state representation, whichmay include a current speed 316. The current image may also include anoverlay 318 on the portion of the road directly in front of the ego-carthat may correspond to a typical lane-width.

The illustrations of an exemplary system in FIGS. 2 and 3A-C demonstratethe utility of certain disclosed systems and methods in a drivingsituation characterized by a curved road with multiple lanes but withouttypical lane markings. In this example, the road had been recently pavedand the lane markings had yet to be painted at the time that this videoframe was captured. Instead, the road has a number of temporary lanemarkings, 322. In this image, 302, a first car 320 appears to bedirectly in front of the ego-car. The mapping of the determined path oftravel 308, however, indicates that the first car 320 is not in the samelane as the ego-car, despite its position directly in front of theego-car. Instead, the determined future path of travel 308 movesrightward before it comes into contact with the first car 320,indicating that the first car 320 is in an unmarked lane to the left ofthe ego-car. The spacing between the future path of travel of theego-car 308 and the current position of the first car 320 can also beseen in the overhead view illustrated in FIG. 3B.

FIG. 4 illustrates an example of mapping a path of travel in accordancewith certain aspects of the present disclosure. A satellite image 400depicts a portion of a curved road. The pin marker 402 indicates thelocation of an ego-car at a first time, time to. The pin marker 404indicates the location of the ego-car at time t1 (t0+1 second). The pinmarker 406 indicates the location of the ego-car at time t2 (t0+2seconds). The pin marker 402 may be referred to as the “current”position in accordance with certain aspects of the present disclosure.The line 408 indicates the path of travel of the ego car from the firsttime, time t0, to a second time, which may be several seconds later.

In FIG. 5A, a mapping of the future path of travel 508 (corresponding tothe path of travel 408 illustrated in FIG. 4 ) is overlaid on the“current” frame 502. The “current” frame 502 corresponds to the position402 of the camera within the ego-car at time t0. The mapping comprises aprojection of the GPS path 504, illustrated in FIG. 5C, from ahorizontal plane 506, to an overhead (bird's-eye-view) reference frame510 illustrated in FIG. 5B, and then to the camera view 502. The bird'seye view reference frame 510 is anchored to the camera position at thebottom. The future path of travel 508, illustrated in FIG. 5B may beprojected into the “current” frame 502 with an inverse perspectivemapping.

FIG. 5A further illustrates that a system may include timestamps for thecurrent video frame 512 and timestamps for the current GPS reading 514.The system may also include an internal state representation, which mayinclude a current speed 516. The current image may also include anoverlay 518 on the portion of the road directly in front of the ego-carthat would correspond to a typical lane-width 518.

The illustrations of the exemplary system in FIGS. 4 and 5A-Cdemonstrate the utility of certain disclosed systems and methods in adriving situation characterized by a curved road with parked cars on oneside. In this example, certain aspects of the disclosed methods help todifferentiate between objects in the path of travel versus outside thepath of travel. In the current frame 502, a central dividing lane marker520 is clearly visible. However, there are no lane markings on the rightside of the ego-car's lane. In this example, the rightmost boundary ofthe ego-car's lane may be inferred from the determined path of travel508, which is at a safe distance from the lane of parked cars. Thespacing between the future path of travel of the ego-car 508 and the rowof parked cars can also be seen in the overhead view illustrated inFIGURE

The “Back-to-the-Future” approach, which is described above for pathprediction, may be used for other aspects of an IDMS as well. Forexample, the IDMS may measure certain driving metrics includingfollowing distance, tailgating, and pedestrian interactions, amongothers. Furthermore, an IDMS based on certain aspects of the presentdisclosure may predict that a vehicle will leave its lane of travel(possibly without signaling/unintentionally), may predict a lane changemay predict a turn, may predict a collision with an object (car,pedestrian, animal, etc.), may predict a change in speed (such as anacceleration or stop), among others. In some embodiments, an enabledsystem may predict driver distraction.

Determining and Mapping a Path of Travel

Several means for determining the future path of travel arecontemplated. These include using a GPS lat/long/altitude trace (e.g. afirst and a second GPS measurement), using future frames withsimultaneous localization and mapping (SLAM) or visual odometry, usingaccelerometer, gyro, and/or magnetometers to dead reckon the path, orusing a combination of these approaches to improve robustness andaccuracy of the future path. Additionally, the GPS location may becombined with map data to refine or validate the determined path. Forexample, the GPS locations may be repositioned to a valid road locationbased on map data to account for GPS inaccuracies. In addition, Lidarand/or radar data may be combined with Lidar and/or radar map data torefine or validate the determined path.

Once the future path is determined, the path and/or object locations maybe mapped into a common setting, or reference frame, to determineinteractions. In one embodiment, the future path may be mapped into aBird's Eye View (BEV) corresponding to an Inverse Perspective Model(IPM) from the camera view to the BEV. Examples of this mapping areillustrated in FIG. 3B and FIG. 5B. After this mapping, a computedhomography matrix may then be used to map the path from the IPMreference frame to the first camera view, which may be referred to asthe camera view reference frame, at time to. Once the determined futurepath is mapped into the camera view reference frame at time t0, thefuture path interactions with objects in the camera view may becalculated.

Alternatively, future path interactions may be calculated in the BEVreference frame. In an embodiment of certain aspects of the presentdisclosure, the camera view pixel locations may be mapped to a BEVreference frame using an IPM and applying the inverse homography matrix.The future path of travel may also be mapped into the BEV interactionsbetween the future path of travel and objects that were mapped from thecamera view may be calculated in the BEV reference frame.

In some embodiments, the homography matrix may be computed based on acamera calibration. In some embodiments, the homography matrix may beauto-calibrated, for example, by using a lane detection method todetermine lane locations and vanishing points in a camera referenceframe. In this example, a lane may be detected and a known lane widthmay be used to calibrate the homography matrix calculation. Similarly,the camera view may be mapped to IPM-view equivalency points, and acorresponding homography matrix may be computed from these mappedpoints. In addition, a homography matrix may be determined usinginertial sensors (such as gyrometer, accelerometer, and barometer),Lidar and/or radar, or GPS to estimate the camera's orientation withrespect to the ground plane.

The future path may be determined in a reference frame that is not tiedto a camera view. For example, a GPS sensor may be used to determine apath of travel in world coordinates. In these embodiments, the futurepath may be oriented (e.g. rotated) so that it corresponds with aspecified camera view. For example, if a camera is mounted in a carwindshield so that it faces forward, a straight and forward pathoriented to the camera view will be a vertical line in the center of thecamera view. Several ways of orienting the future path are contemplated.In one embodiment, the direction of travel may be measured. Ameasurement of the direction of travel may be computed using GPS data,for example. In another embodiment, the direction of travel may beassumed to be forward, so that the instantaneous path of travel isstraight and forward. In this example, the path tangent at time t0 maylie straight ahead.

When the path is straight and forward from the camera view, the path inthe IPM-based reference frame may include a first and a second datapoint that may be mapped to the y-axis straight ahead of the camera, inwhich y is the up-down axis of the camera view. For a camera mounted toa car, changes in the y-direction of the future path that are notaccompanied by changes in the x-direction may indicate a path of travelthat is straight and forward.

In some embodiments, the data points may be curve fit, or interpolated,to determine a tangential path at time t0 for orienting the path. Inaddition, a magnetometer (such as a compass) may be accessed todetermine the compass orientation of the vehicle at time to. In thisexample, the reading from the magnetometer may be oriented to align theGPS track according to the magnetometer information. In this example,the orientation may be further refined based on a known or estimatedmagnetic deviation for the geographic lat/long location.

In some embodiments, GPS data may be used to compute the future path oftravel. In one example, the lat/long values may be used to determine a2D path. This approach may yield acceptable results if the road forms aground plane. In another example, the altitude information may be usedto enhance the determined future path of travel to include a 3D paththat may account for inclines. In addition, inertial sensors may be usedto refine an estimate of the angle of inclination.

Use of “Back-to-the-Future” Methods in Learning Systems

It may be desirable for a learning system to learn to predict futurepaths of travel. A challenge for learning systems, however, may be arequirement for labeled data. For example, labeled data may be a videoframe for which a person has labeled boundaries of a visible lane and/oran appropriate future driving path given the video frame. In some cases,an appropriate future driving path may depend on a state of the vehicle,which may include a driving speed. In some cases there may be multipleappropriate paths of travel. Alternatively, or in addition to labeleddata with respect to a video frame, labeled data may include a satelliteimage for which a person has labeled boundaries of a visible lane and/oran appropriate future driving path of travel. To effectively learn topredict paths of travel, a learning system may require an amount oflabeled that would be burdensome or infeasible for a human labeler or agroup of human labelers to produce.

Certain aspects of the present disclosure are directed to methods thatmay enable training a learning system on data for which some videoframes may not contain human provided labels. In some cases, videoframes with human provided labels may also be used. The application ofthese methods may enable a learning system so that it becomes capable ofpath of travel prediction. In addition, the application of certaindisclosed methods may decrease overall training time compared tolearning systems that rely only on human-labeled data. By utilizing thedisclosed methods, a system may be created that is predictive of a pathof travel.

In the “Back-to-the-Future” approach, a path “prediction” may becompared with an accurate prediction of the future, which may beachieved by actually looking into the future, as described above. Whenapplied to predicting and training, the “Back-to-the-Future” approachmay be used as a measured future against which to train a machinelearning algorithm's predicted future. In this training example, themachine-learning algorithm may not have access to the future data duringits prediction step. Rather, the future data would serve as a type ofmachine-labeled data set against which to compare the prediction. Inthis way, the future data may be used as “labels” which may largelyreplace or diminish a need for human labeling to indicate a likely path.

In an embodiment of the present disclosure, a “Back-to-the-Future”method may be used, such that a future path of travel starting at agiven time may be determined. In this example, the need for humanlabeling or interaction may be significantly reduced or eliminated. Amachine learning engine, which may include a deep learning engine, maybe trained to predict the future path given a current frame. In someembodiments, the learning engine may also have the current and recentpast inertial data and/or GPS data tracks as inputs, and/or the pastseries of frames. In addition, inputs to the learning engine may alsoinclude a current internal state estimate. An internal state estimatemay include, for example, a current velocity, extrinsic or intrinsiccamera parameters, and the like.

In this example, the data available to the deep learning engine mayinclude data recorded up until a given point in time (which may bereferred to as a first time), which may include an internal stateestimate. The deep learning engine may generate a prediction based onthese inputs. In some embodiments of the present disclosure, the deeplearning engine may refine its parameters, or learn, based on thedifference between this predicted path of travel and the subsequentlydetermined future path of travel. The determined path of travel may bethe path from the first time to a later second time.

In one embodiment, the input to a deep neural network may be a singleframe, and the output may be a predicted path. The predicted path may bea direction-normalized path in an IPM-based reference frame. In anotherembodiment, the input may be a sequence of a number of past and currentframes, and the output may be a predicted path. In some embodiments, theinput may be a sequence of frames and the network architecture mayinclude a feed-forward structure for which a group of frames taken froma window of time are presented together to the network as inputs. In analternative embodiment, the input may be a group of frames that arepresented sequentially to a neural network that has a recurrent neuralnetwork structure. In these examples, the output of the feed-forward orrecurrent neural network may be a prediction of the future path oftravel. Other embodiments of a path prediction engine in accordance withcertain aspects of the present disclosure may utilize additional oralternative inputs from inertial sensors, on-board diagnostic (OBD)inputs such as speedometer, fuel flow, and additional sensors.

In one embodiment, at any given time t, an Artificial Intelligence (AI)engine may predict a path of travel, p, where p is a vector p(t),p(t+dt), p(t+2dt) . . . (p+(N−1)dt). Over the next Ndt time instances,the system may measure the actual path of travel a, and then calculatethe error |a−p|. The frame of reference may shift over time as thevehicle moves, but the determined actual path of travel, a, may beprojected, or mapped, back in time to a_hat, which may correspond to thetime t. Alternatively, p, may be mapped to a newer frame of reference asp_hat.

In this example, p and a may be functions of road conditions, vehicletype, vehicle speed, other inertial data, traffic situation, trafficsigns, GPS readings, environment, region, lane markings, and the like.

For training, it may be assumed that most users of a system thatincorporates certain aspects of the present disclosure are “good”drivers. By aggregating data from a diversity of users and in a varietyof conditions, a learning system configured according to certain methodsof the present disclosure may learn to accurately predict future pathsof travel based on currently available data. Such a system may then beapplied in ADAS or autonomous driving systems for which real-time ornear-real time performance may be desired. For example, the system couldissue a predicted path of travel as a target path for the control systemof an autonomous car. Because a “Back-to-the-Future” approach mayautomatically produce a predicted path of travel that may closelyapproximate an actual path of travel, a, in an efficient manner, alearning system based on such data may learn to “drive” in types ofdriving conditions that remain challenging for current ADAS andautonomous driving systems by supplying the requisite amount ofmachine-labeled data.

A system that determines a future path of travel in accordance withcertain aspects of the present disclosure may have many applications.For example, a system in accordance with certain aspects of the presentdisclosure may be employed in an AI engine of an IDMS, an ADAS, andautonomous driving, or a driver monitoring application. In an IDMS, forexample, a desired latency may be less than that of an ADAS andautonomous driving system. As a consequence, the training of apredictive AI engine, which may be applied to an IDMS, driver monitoringsystem, ADAS, autonomous driving system, and others, need not occur inreal-time or near real-time. Rather, the system could store the p and avectors in a memory. In some embodiments, the stored p and a vectors maybe accessed at a later time and presented to a learning system in arandomized fashion. A randomized presentation of inputs may result in animproved learning behavior. In some embodiments, the training engine maybe processed on a local processor. This may be used to personalize asystem to a particular driver. Alternatively, the training engine may beprocessed on one or more processors in the cloud. In this example, thedevice that is configured according to certain aspects of the presentdisclosure may compute sampled p and a vectors on a local processor andsend the computed vectors to the cloud for training. In someembodiments, the “Back-to-the-Future” approach may be applied to datathat were collected in a driver monitoring, ADAS, or autonomous drivingsystem, among others.

Additional sensors may be provided as inputs to a learning system inaccordance with certain aspects of the present disclosure. Additionalsensor inputs may include additional cameras, Lidar, radar,environmental sensors, Map data, other information from external sources(internet, other cars, etc.), and others.

A learning system in accordance with certain aspects of the presentdisclosure may update the parameters of an AI engine based on predictionerrors, which may be a difference between predicted and actual drivingbehaviors. Such a prediction error may not represent an error by the AIengine in some cases. For example, the driver may exhibit poor drivingbehavior over the course of the predicted path. In addition, the drivermay leave the current ‘lane’ to follow a certain pre-determined route oftravel. In addition, the driver may initiate a discretionary lanechange, may stop to take a break from driving, or may take an unplanneddetour. In these cases, the actual path of travel may not be a goodproxy for the predicted path of travel. If these errors in theprediction error are rare (as they may be expected to be) then thesystem may learn to ignore them. The system may also proactivelydecrease the impact of these kinds of errors. Several methods fordecreasing these types of errors are contemplated, including: (a) Detectand ignore large errors (i.e. outlier detection). An example would be toignore errors beyond a certain threshold; (b) Ignore feedback fromdrivers with a poor driving score over the last T seconds where T is aconfigurable parameter; (c) Use a pre-programmed GPS route informationto modify the predicted path of travel.

Since training may not occur in real-time or near real-time, a systemmay store all of the actual and predicted paths, or a sample of theactual and predicted paths, and make a determination on which “actuals”to use for calculating the prediction error at a later time. In oneexample, the criteria for using the “actuals” may include anidentification of at least one of the conditions mentioned above, whichmay flag the data vector for exclusion. Alternatively, the system maymodify the predicted path post-hoc to account for the outlier actuals.For instance, once the system detects that a lane change occurred attime to, the predicted path starting from time t0 could be modified toaccount for a lane change.

According to certain aspects of the present disclosure, a machinelearning system may be trained. Based on a reference state, s, theobject's predicted path of travel, p, may be computed, where p is avector p(t), p(t+dt), p(t+2dt) . . . (p+(N−1)dt. In some embodiments,p(t) may be the controllable vehicle state at time t. In this example,controllable vehicle state may refer to parameters of the vehicle'sstate that can be controlled by the vehicle. This may include thevehicle's GPS location, speed, lane position, accelerometer, etc., butmay exclude parameters such as traffic condition, weather condition,etc., that are outside the control of the object.

Based on the current state of the vehicle, s(t0), a desired path oftravel may be computed and used for training purposes as the “actual”path of travel, a, where a is the same vector type as p. Several meansfor computing a are contemplated. In one embodiment, objects in theactual path of travel may be mapped to the current frame of reference.In another embodiment, crowd sourced data from multiple vehicles who are(or were) in a similar state may be used. Several means for using thecrowd sourced data are contemplated. The crowd sourced data may becombined, averaged, and/or clustered to compute a desired path oftravel. In this example, the “actual” path of travel may be considered anormal or typical course of action from a given state that is based onprevious recordings of moving objects in the same or similar states. Aprediction error may be calculated based on the predicted and desiredpath of travel, and further, may be used to train a learning system. Insome cases, there may be multiple predicted paths of travel.

Several methods for improving accuracy of a learning system inaccordance with certain aspects of the present disclosure arecontemplated. These include, but are not limited to, outlier detection,use of crowd sourced data selected from drivers with a “good” drivingscore, use of GPS route information, among others.

For a learning system, a “predicted path of travel” may be considered apredicted time sequence. In addition to using certain disclosed methodsto train an AI system to predict a path of travel, the disclosedlearning methods may be used to train lane detection, and other types ofdriver behavior including but not limited to: traffic sign compliance,pedestrian interactions, tailgating, and adapting to road and weatherconditions.

Several uses for predictions from a trained machine learning algorithmare contemplated. For example, a system may predict what a good humandriver would do in a given circumstance. An autonomous car may then,according to certain aspects of the present disclosure, learn to performdriving patterns that are more human-like. An issue with currentautonomous vehicles is that they drive in non-human ways. For example,current autonomous vehicles may not exhibit expected “non-verbal” or“gestural” cues of their intended actions. The lack of such human-likedriving behavior may result in more rear-end collisions. The“Back-to-the-Future” approach, therefore, may improve “non-verbal” or“gestural” signals of an autonomous vehicle.

In some embodiments, a system in accordance with the present disclosuremay constantly or periodically record the path of travel and may thendetermine an “actual” path of travel based on one or more trainervehicles that have been in a similar state, s. A “trainer” vehicle maybe any vehicle that can serve as a reference for training. Trainervehicles may include human driven vehicles as well as vehicles that aredriven by other means. For example, an autonomous vehicle with asuperior driving performance may be used as a trainer vehicle for anautonomous vehicle that may be less expensive, or that may have lessdriving experience. Any vehicle that is expected to exhibit betterdriving behavior than the current system may be used as a trainervehicle. Based on the vehicle's current state (which may include GPS,speed, weather conditions, vehicle type, etc.) the system may determinethe “actual” path of travel (which may be a “target” path or a “desired”path) from one or more “trainer” vehicles with a similar state, s.

In some situations, a vehicle may have multiple “correct” paths tochoose from. For example, a vehicle may change lanes, or may not changelanes, at a given point in its journey, and either choice may have beenan appropriate driving path. When there are multiple “correct” paths oftravel, the determination of a “correct” actual path of travel mayinclude selecting from among the available choices. In some cases, thechoice of one correct path among several possible correct choices maynot impact the key performance indicators of the driving algorithm in,for example, an IDMS or in a machine learning context. In cases withmultiple “correct” paths of travel, an AI engine could generate multipledifferent predicted paths of travel.

In a situation in which there is more than one actual path of travel,the training of an AI engine in accordance with certain aspects of thepresent disclosure may include selecting one of the predicted paths oftravel. For example, the path that is closest to the actual path oftravel may be selected according to an objective cost function. Aprediction error may be determined between the selected path and thedetermined actual path. In some embodiments, the prediction error may beused that to improve the performance of the AI engine, which may includethe AI engine's path prediction performance.

In an embodiment in which the AI engine is using the actual path oftravel of the vehicle, and not using a method that includes crowdsourcing, the AI engine may only have access to the one actual path oftravel. That is, the AI engine may only have access to the actual pathof travel of the vehicle according to the “Back-to-the-Future” approachdescribed herein. In contrast, in an embodiment in which the AI engineis using crowd-sourcing to predict a path of travel from previouslyrecorded actual paths of travel from multiple vehicles, a system inaccordance with the present disclosure may generate a number, K, ofdesired paths of travel as a function of the current state. In thiscase, the system may select a subsample of the predicted paths of travelto compare against the K desired paths of travel. The system may thenselect one path of travel according to an objective cost function. Thesubsample, K, of desired paths of travel may include all of the paths,M, that were predicted based on a crowd-sourcing method.

While some of the examples presented above relate to vehicles, aspectsof the present disclosure may be applied to any AI system for which ismay be desirable to predict a future course of action based on currentstate. In one embodiment, a reference or trainer system may be providedby humans performing the same task. For example, aspects of the presentdisclosure may be applied to a Robot for which an ability to navigateinside a building may be desirable. Humans equipped with similar sensorsas the Robot may serve as the reference “trainer” system. The human's“actual” path of travel may be projected or mapped onto the Robot'scurrent frame of reference. In some embodiments, a learning system maycompare the actual path of travel with the Robot's predicted path oftravel. This comparison may be used to teach the Robot what to do next.In addition to helping the Robot learn to navigate a building, certainaspects of the present disclosure may be applied to learning otheruseful behaviors, such as how to avoid obstacles, climb stairs, etc.

Certain aspects of the present disclosure may be applied to anautonomous drone that is learning to fly (navigate, avoid obstacles, docartwheels, etc.). In this example, the reference, or trainer, systemcould be a drone operated by a human, for example.

Distant Object Detection

For IDMS, driver monitoring, ADAS, autonomous driving systems, and thelike, it may be desirable to detect distant objects that may beencountered while driving, including cars, pedestrians, bicyclists,motorcyclists, and other objects. For heavy commercial vehiclestraveling at freeway speeds, drivers may be encouraged to be aware ofvehicles out to 12 seconds, or more than 1200 feet. Visual detection ofcars and other objects out to these distances for a desirable camerafield of view (FOV) may require a high-resolution capture.

Current systems may have some object detection capabilities, but theprocessing speed and accuracy of these systems often has a tradeoff. Forexample, processing high-resolution images in detail may provideimproved accuracy for identifying objects, particularly for more distantor smaller objects, but for current systems the higher resolution comeswith an increased processing cost in terms of additional pixels toprocess. Accordingly, there is a need for methods to quickly determineobject locations at far distances, including objects that may impact avehicle's path of travel. Certain aspects of the present disclosure aredirected to improved methods for detecting objects.

Current object detection methods may include processing an entire imageat one scale, or processing an entire image at multiple scales using aspatial pyramid. A Region of Interest (ROI) may be used to select asub-region of the image. The ROI may be processed at the same set ofscales as would be applied to the entire image. Processing ofhigh-resolution images, such as resolutions of 1080P, 4K, or higherresolution may be prohibitive in terms of computational cost. Tomaintain real-time or near real-time performance, object detectiondevices may reduce frame rates or may require additional computationalunits at increased cost in money, power, space, and heat. Objectdetection at a distance, therefore, may not be feasible for mobilereal-time in-device computation with consumer grade parts. In somecases, a tradeoff may be chosen to process images at lower resolution,but such a system may only reliably detect objects such as cars out to afew hundred feet or less.

Certain aspects of the present disclosure are directed to improvedmethods of object detection that may enable detection of objects such ascars at a greater distance than would be possible by a comparablyequipped system that do not include the disclosed methods. In oneembodiment, multiple regions of interest (ROI) may be determined in avideo frame or sequence, and different ROIs may be processed atdifferent resolutions. In one embodiment, a determined region ofinterest may be associated with the driving or road environment, and maycut off the regions of the image corresponding to the sky and tops ofhigh rise buildings to reduce the area of computation. The detectionalgorithm may process the initial ROI at a low resolution, such as 480p,or potentially lower. An appropriate “low resolution” may be higher astechnology advances. In this ROI, one or more sub-ROIs may bedetermined, and the detection algorithm may be run at successivelyhigher resolutions in these sub-ROIs. This method may enable processingthe entire frame, but targeting the computationally expensive highresolution processing for small ROIs around the region of interest thatare more likely to have distant objects, while processing more of theframe at the less computationally expensive lower resolution, or not atall.

Several methods for determining the sub-ROIs are contemplated. In oneembodiment with a single far field ROI, the far field ROI may bedetermined statically at install or calibration time of a device that isconfigured according to certain aspects of the present disclosure. Thefar field ROI may be stored as a set of parameters. In anotherembodiment, the far field ROI may be relatively static, but determinedusing on-line calibration such as by determining the vanishing pointbased on lane markings and creating a rectangular sub-ROI around thevanishing point.

In another embodiment, the one or more far field ROIs may be determinedbased on the predicted path of travel. In one embodiment, the sub-ROImay be determined as a region around the area where the lanes end or thepath of travel ends in the image, so that it adapts to curves in theroad; this may allow for a smaller sub-ROI than a static or relativelystatic ROI. In a more targeted sense, for the purposes of determiningfollowing distance, for example, the sub-ROI may be restricted to thelane or a fixed width around the path of travel.

In another embodiment, the sub-ROI may additionally be based on theestimated distance of the sub-ROI. This estimated distance may be basedon the measured width of the lane markings, or based on the computed GPSdistance from a back-to-the-future path estimate. The resolution usedfor each sub-ROI may be based on the estimated distance targeting aminimum pixel density such as pixel width or height of objects. Forexample, a device that embodies certain aspects of the presentdisclosure may target a minimum of 20 pixels assuming a typical carwidth of 1.8 m and accounting for the camera FOV in pixels.

Detecting objects at long distances may be difficult, as distant objectsmay appear small and may occupy a small amount of area in a video frameor in a Lidar or radar scan. Accordingly, an embodiment of the“Back-to-the-Future” approach may be applied to distant objectdetection. Objects, such as cars or pedestrians, may be detected in a“future” frame when they are closer to an ego-car and in which they mayappear larger and may occupy a larger amount of area in a video frame orin a Lidar or radar scan. The object that is detected in the “future”may be then tracked backward in time to an earlier frame. Trackingtechniques such as HOG or motion vectors may be used to track the objectacross frames. In this way, an object may be detected, and may beidentified in an earlier frame for which the detection would have beenchallenging if the “Back-to-the-Future” approach were not available.

As with determining the path of travel, the object detection predictionsbased on the earlier frames may be combined with the determined objectdetection from the “Back-to-the-Future” approach, to train learningsystems with minimal or no human labeling. In this embodiment, abounding box and HOG or other tracking features may be determined in thefuture close-up frame, and tracked backward in time to the earlier frameusing the later time frame to label and establish a bounding box on theearlier frame. This may also improve the accuracy of labels on objectsthat may be hard to distinguish otherwise.

The location of objects may be useful, for example, in determiningwhether a path of travel will bring a car into close proximity of adetected object. For example, object detection may be a desirablefeature that could be used in conjunction with a path predicting and/orpath determining system, such as one using a “Back-to-the-Future”approach or a predictive system approach that was trained on“Back-to-the-Future” labeled data in accordance to certain aspects ofthe present disclosure, or may be used in conjunction with computervision based lane detection approaches, or other approaches.

Determining Vehicle Interactions

Certain aspects of the present disclosure may be applied to systems thatdetermine driving behaviors in response to objects in the environment.Tailgating, for example, is a driving behavior in which a car drives atan unsafe distance behind the car ahead of it in the same lane. Since adetermination of tailgating behavior depends on a determination of thelane occupied by two cars, a method for determining the lane of travelmay be desirable for a driver monitoring system. A driving monitoringsystem with robust lane detection, for example, may be able to properlyignore situations that may appear to be tailgating but in which the carahead is actually in a different lane. As described above, FIG. 2illustrates an example of what may appear to be tailgating behavior, butin which the car ahead is actually in an adjacent lane and travellingaround a curve.

Given the determined object locations and determined future path oftravel, either estimated or measured and projected, it may be desirableto determine interactions between the determined path and the detectedobjects. Approaches that estimate a path of travel or measure the futurepath of travel may assume a path width based on either a typical lanewidth, such as 3.7 m, or based on the measured or estimated typicalwidth of the ego-car. Additionally, the future path may use laneestimation so that when lane markings are present the width and shift ofthe lane markings may be adaptively matched up to the computed orestimated future path of travel. In this example, when the vehicletravels in areas where the lane tracking algorithm temporary losestrack, such as in areas with lighting variations from overpasses, orareas facing toward the sun at sunrise/sunset, or fresh pavement wherelane markings have not yet been added, the path of travel alone maydetermine the lanes. Still, the determined location and width of thelanes may be more accurate due to the recent prior interaction with thelane detection algorithm. In some embodiments, a system may measure theoffset from the center lane and the width of a lane. Alternatively, thesystem may include a tracking algorithm such as a Kalman Filter.

If the measured path of travel is based on GPS, accuracy for a cameramay be improved based on radar or Lidar. In this example, a projectedGPS path of travel in an IPM reference frame may be overlaid with theradar or Lidar data to determine if detected objects lie in the path oftravel, even if they lie on curved roads.

A hybrid system is contemplated in which an inference engine provides aconfidence level on a prediction error computed in “real-time” based oncurrent inputs and is compared with a confidence level based on a“Back-to-the-Future” approach in accordance with certain aspects of thepresent disclosure. The system may choose to use one approach over theother based on the confidence intervals. For a road with clear lanemarkings in a hilly area, for example, the inference engine may detectthe lanes with high confidence, but may have low confidence inprojecting the future into the current frame of reference due to poorestimates of GPS, elevation changes, and the like.

As used herein, the term “determining” encompasses a wide variety ofactions. For example, “determining” may include calculating, computing,processing, deriving, investigating, looking up (e.g., looking up in atable, a database or another data structure), ascertaining and the like.Additionally, “determining” may include receiving (e.g., receivinginformation), accessing (e.g., accessing data in a memory) and the like.Furthermore, “determining” may include resolving, selecting, choosing,establishing and the like.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: a, b, or c” is intended to cover: a, b, c,a-b, a-c, b-c, and a-b-c.

The methods disclosed herein comprise one or more steps or actions forachieving the described method. The method steps and/or actions may beinterchanged with one another without departing from the scope of theclaims. In other words, unless a specific order of steps or actions isspecified, the order and/or use of specific steps and/or actions may bemodified without departing from the scope of the claims.

The processing system may be configured as a general-purpose processingsystem with one or more microprocessors providing the processorfunctionality and external memory providing at least a portion of themachine-readable media, all linked together with other supportingcircuitry through an external bus architecture. Alternatively, theprocessing system may comprise one or more specialized processors forimplementing the neural networks, for example, as well as for otherprocessing systems described herein.

Thus, certain aspects may comprise a computer program product forperforming the operations presented herein. For example, such a computerprogram product may comprise a computer-readable medium havinginstructions stored (and/or encoded) thereon, the instructions beingexecutable by one or more processors to perform the operations describedherein. For certain aspects, the computer program product may includepackaging material.

Further, it should be appreciated that modules and/or other appropriatemeans for performing the methods and techniques described herein can bedownloaded and/or otherwise obtained by a user terminal and/or basestation as applicable. For example, such a device can be coupled to aserver to facilitate the transfer of means for performing the methodsdescribed herein. Alternatively, various methods described herein can beprovided via storage means (e.g., RAM, ROM, a physical storage mediumsuch as a compact disc (CD) or floppy disk, etc.), such that a userterminal and/or base station can obtain the various methods uponcoupling or providing the storage means to the device. Moreover, anyother suitable technique for providing the methods and techniquesdescribed herein to a device can be utilized.

It is to be understood that the claims are not limited to the preciseconfiguration and components illustrated above. Various modifications,changes and variations may be made in the arrangement, operation anddetails of the methods and apparatus described above without departingfrom the scope of the claims.

What is claimed is:
 1. A method of processing a camera frame in a mobiledevice, the method comprising: capturing the camera frame using a cameramounted on a vehicle traveling on a road; receiving the camera framefrom the camera; determining a drivable path of the vehicle; projectingthe drivable path onto the camera frame; partitioning a part of thecamera frame containing the drivable path into at least one sectionbased on a distance from the vehicle to each of the at least onesection; and determining a resolution for processing of each of the atleast one section based on the distance from the vehicle to each of theat least one section.
 2. The method of claim 1, further comprising:determining the drivable path by using a map of the road.
 3. The methodof claim 1, further comprising: determining the drivable path of thevehicle by using a sensor on the vehicle.
 4. The method of claim 1,further comprising: determining a section of the camera frame that doesnot contain the drivable path.
 5. The method of claim 4, furthercomprising: discarding the section of the camera frame that does notcontain the drivable path.
 6. The method of claim 1, further comprising:processing information in each of the at least one section using thedetermined resolution for processing.
 7. The method of claim 3, furthercomprising: determining a position of the vehicle, an orientation of thevehicle or a velocity of the vehicle by using the sensor.
 8. The methodof claim 7, wherein the drivable path of the vehicle is determined basedon the position of the vehicle, the orientation of the vehicle or thevelocity of the vehicle.
 9. The method of claim 3, wherein the part ofthe camera frame containing the drivable path is partitioned into the atleast one section based on a condition of the road.
 10. A devicecomprising: a memory; and a processor communicatively coupled to thememory, the processor configured to: receive a camera frame from acamera mounted on a vehicle traveling on a road; determine a drivablepath of the vehicle; project the drivable path onto the camera frame;partition a part of the camera frame containing the drivable path intoat least one section based on a distance from the vehicle to each of theat least one section; and determine a resolution for processing of eachof the at least one section based on the distance from the vehicle toeach of the at least one section.
 11. The device of claim 10, whereinthe processor is configured to determine the drivable path by using amap of the road.
 12. The device of claim 10, wherein the processor isconfigured to determine the drivable path of the vehicle by using asensor on the vehicle.
 13. The device of claim 10, wherein the processoris configured to determine a section of the camera frame that does notcontain the drivable path.
 14. The device of claim 13, wherein theprocessor is configured to discard the section of the camera frame thatdoes not contain the drivable path.
 15. The device of claim 10, whereinthe processor is configured to process information in each of the atleast one section using the determined resolution for processing. 16.The device of claim 12, wherein the processor is further configured todetermine a position of the vehicle, an orientation of the vehicle or avelocity of the vehicle by using the sensor.
 17. The device of claim 16,wherein the drivable path of the vehicle is determined based on theposition of the vehicle, the orientation of the vehicle or the velocityof the vehicle.
 18. The device of claim 12, wherein the processor isfurther configured to partition the part of the camera frame containingthe drivable path into the at least one section based on a condition ofthe road.
 19. A non-transitory computer-readable storage mediumcomprising code, which, when executed by a processor, causes theprocessor to process a camera frame in a device, the non-transitorycomputer-readable storage medium comprising code for: capturing thecamera frame using a camera mounted on a vehicle traveling on a road;receiving the camera frame from the camera; determining a drivable pathof the vehicle; projecting the drivable path onto the camera frame;partitioning a part of the camera frame containing the drivable pathinto at least one section based on a distance from the vehicle to eachof the at least one section; and determining a resolution for processingof each of the at least one section based on the distance from thevehicle to each of the at least one section.
 20. The non-transitorycomputer-readable storage medium of claim 19, further comprising codefor determining the drivable path by using a map of the road.
 21. Thenon-transitory computer-readable storage medium of claim 19, furthercomprising code for determining the drivable path of the vehicle using asensor on the vehicle.
 22. The non-transitory computer-readable storagemedium of claim 19, further comprising code for determining a section ofthe camera frame that does not contain the drivable path.
 23. Thenon-transitory computer-readable storage medium of claim 22, furthercomprising code for discarding the section of the camera frame that doesnot contain the drivable path.
 24. The non-transitory computer-readablestorage medium of claim 19, further comprising code for processinginformation in each of the at least one section using the determinedresolution for processing.